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Abstract — One-shot achievable rate region for source cod- 
ing when coded side information is available at the decoder 
(source coding with a helper) is proposed. The achievable 
region proposed is in terms of conditional smooth max Renyi 
entropy and smooth max Renyi divergence. Asymptotically 
(in the limit of large block lengths) this region is quantified in 
terms of spectral-sup conditional entropy rate and spectral- 
sup mutual information rate. In particular, it coincides with 
the rate region derived in the limit of unlimited independent 
and identically distributed copies of the sources. 

I. Introduction 

The derivation of most of the fundamental results in 
information theory relies on the assumption that a random 
experiment is repeated identically and independently for 
large enough time. However, in practical scenarios both of 
these assumptions are not always justifiable. To overcome 
the limitations posed by these assumptions Renner et 
al. introduced the notion of one-shot information theory. 
One-shot information theory relies on the fact that a 
random experiment is available only once. Thus removing 
both the assumptions together. 

The first one-shot bounds were given for the task of 
one-shot source coding JTJ . These bounds were based 
on smooth Renyi entropies. The notion of smooth Renyi 
entropies were introduced for the very first time in the 
same work, i.e., in Ref. JT]. The elegance of the one- 
shot bounds obtained in Ref. HI is that these bounds 
coincide with the Shannon entropy J2] of the information 
source in the limit of unlimited independent and identi- 
cally distributed (i.i.d.) copies of the source. Furthermore, 
these bounds coincide with spectral sup-entropy rate as 
defined by Han and Verdu in Ref. J3 in the limit of 
unlimited arbitrarily distributed copies of the source. One- 
shot bounds for distributed source coding were given by 
Sharma et al. in 2). In Wang et al. gave one-shot 
bounds for the channel coding problem in terms of smooth 
min Renyi divergence. 

There has been a considerable work on the one-shot 
bounds for the quantum case under various scenarios 
(see for example Refs. 0, Q, 0, 0, Q0), fll] and 
references therein). 



In this work we give one-shot achievable rate region for 
source coding when coded state side information is avail- 
able at the decoder. The achievable rate region derived for 
this problem is in terms of smooth max Renyi divergence 
and conditional smooth max Renyi entropy. The notion of 
smooth max Renyi divergence was introduced by Datta 
for the quantum case in lfl2l . We further show that the 
achievable region obtained asymptotically coincides with 
the rate region derived in ifPJl . 

The rest of this paper is organized as follows. In 
Section [TT] we discuss the notations which we will be 
using throughout this paper. In Section |TlT] we give the 
definitions of of smooth conditional Renyi entropy of 
order zero and smooth max Renyi divergence. We then 
prove two lemmas pertaining to the properties of smooth 
max Renyi divergence. Although, the proof of Lemma [3] 
is known in the quantum case we give a totally different 
proof. In particular, our proof involves more straight 
forward arguments. In Section [IV] we state and prove the 
achievable region for source coding problem when coded 
side information is available at the decoder. 

II. Notations 

In the discussions below we will be using X to 
represent a random variable. We will assume that all the 
random variables are discrete and have finite range. We 
represent a random sequence of length n by X n and a 
particular realization of X n by x. Notation X will be used 
to represent an arbitrary sequence of random variables, 
i.e., X = {X n }^ =1 . We use the notation | • | to represent 
the cardinality of a set. The set {x : Px™(x) > 0} is 
denoted by Supp(Px™)- We use the notation 



X -» Y 



Z 



to denote the fact that random variables X, Y and Z 
form a Markov chain. We represent the following set of 
real numbers 

{x : < x < oo} 

by M. + . X x y will represent the cartesian product of 
two sets. Similarly [X x y) n will represent the n-th 



Cartesian product of the set X X y. The notation N is 
used to represent the set of natural numbers. Throughout 
this paper we will assume that log is to the base 2. 

III. Smooth Renyi divergence of order infinity 

AND CONDITIONAL SMOOTH RENYI ENTROPY OF 
ORDER ZERO 

Definition 1: (Max Renyi entropy Q4)) Let X ~ P x , 
with range X. The zero order Renyi entropy of X is 
defined as 

H {X) :=logSupp(P x ). 

Definition 2: (Conditional smooth max Renyi entropy 
ifBI ) Let (X, Y) ~ P XY , with range X x y. For e > 0, 
the conditional smooth Renyi entropy of order zero of X 
given Y is defined as 

H £ JX\Y):= min logmax|Supp(Q(X|Y = y))\, 

where B%P X y) = {Q : E* >v e*xy > 1 - 

e,V (x,y) G X x y, P X Y(x,y) > Q(x,y) > 0} and 
Q(X = x\Y = y):= $2$, for any x G X and y G 3>. 
With the convention that Qpf = x\Y = y) := Oif 
P Y (y) = 0. 

Definition 3: (Max Renyi divergence O) Let P and 
Q be two probability mass functions on the set X such 
that Supp(P) C Supp(Q). The max Renyi divergence 
between P and Q is defined as 



Doo(P\\Q) := log max 



P(x) 



x:P{x)>0 Q(x) ' 

Definition 4: (Smooth max Renyi divergence) Let P 
and Q be two probability mass functions on the set X 
such that Supp(P) C Supp(Q). The smooth max Renyi 
divergence between P and Q for e G [0, 1) is defined as 



^L(^IIQ) :=1 «g inf max 



where 



B £ (P) 



0GB E (P) K:P(a;)>0 Q(x) ' 



< 4>{x) < P(x),Wx £ X and 



Notice that D e OQ (P\\Q) is a non increasing function of e. 

Lemma 1: (Datta and Renner J6)) Let (X, Y) = 
{(X n ,Y n )}^ =1 be an arbitrary random sequence taking 
values over the set {(X x y) n }%Li, where (A" x y) n is 
the n-th Cartesian product of X x y. Then 

lim limsup ! '- = H(X\Y), 

€ >0 n— VrxD Tl 



where 

P(X|Y) := 
infX a I lim inf Pr 



n ° g P X n, y „(X«|Y«) 



< a 



1 



P(X|Y) is called the spectral-sup conditional entropy 
rate of X given Y flTfj). In particular, if (X,Y) = 
{(X n , is a random sequence of independent and 

identically distributed random pairs distributed according 
to Pxy then 

lim limsup H o( Xn \ Yn ) = H(X\Y). 

Lemma 2: Let P and Q be two probability mass func- 
tions defined on the set X, where Supp(P) C Supp(Q) 
and \X\ < oo. There exists G B £ (P) such that 



^(^ll<9)=logmax 



(1) 



c€X Q(x) ' 

Proof: Without loss of generality we assume here 
that X C N. We construct the </>iS by decreasing the PjS 
such that the total decrease is e, i.e., 5Z igi ^(Pj — 4>i) = £■ 
The following procedure achieves the above. 

Step —5- (Initialization) <j>i = Pi Mi G X. 
Step 1 — > Let ri , r 2 be the largest and the second largest 
ratios in 

A=^:iEX^ (2) 

respectively. Let I be the collection of all is 
that have the highest ratio, i.e., 



ri 



(3) 



If |^4| = 1 then notice that I = X. In this case 
start decreasing all (/>;S where i G X such that 
all the indices continue to have constant ratio, 
i.e., q 1 = Vi, j G X. Continue this process 
until we run out of e, i.e., 2~2i£x(^i ~ 00 = e 
in which case end the procedure. Else go to 
step 2. 

Step 2 — > We start decreasing all <friS where i E I such 
that indices in / continue to have the highest 
ratio, i.e., $f = — Vi, j E I. As a result, r\ 
will start decreasing. Continue decreasing till 
either 

Case 1: r\ hits i.e., r\ = T2 in which case stop 
decreasing any further. Goto step 1. Or 

Case 2: we run out of e, i.e., 2~2iei(Pi ~ 4>i) = e in 
which case end the procedure. 



We claim that the <j> constructed by the above procedure 
is such that 



logmax^4=^(P||Q). 



(4) 



x£X Q(x) 

We give a proof by contradiction to prove (125V Let </>' e 
B £ (P) be the output of some other procedure such that 



log max — — > log max — — . 

xex Q(x) xex Q[x) 



(5) 



Let A = {i £ X : 0, < Pi}. Notice that for every 

i,j G i 

Qi Qj 

It is easy to observe that for (0 to hold 4>' must satisfy 
the following 

# < V^ e AT. (6) 

However, this is not possible because this new procedure 
will not have enough e to accomplish ©, i.e., 

$>Wi)> e - 

■ 

Remark: It is easy to observe from the proof of Lemma 
that for e g [0,1), 



Supp(0) = Supp(P), 



(7) 



where <j> satisfies (Q~|). 

Lemma 3: Let P = {P n }n=i and Q = {Qn}n=i 
be an arbitrary sequences of probability mass functions 
defined on the set {X n }%L 1 , where X n is the ro-th 
cartesian product of the set X and \X\ < oo. Assume 
that for every n E N, Supp(P„) C Supp(Q„). Then 



limlimsup-D^,(P n ||Q n ) =I(P;Q), 



£-t0 



(8) 



where 



I(P; Q) := inf I a\ liminf Pr { — log — — < a 



(9) 



I(P; Q) is called the spectral sup-mutual information rate 
between P and Q US). In particular, if P = {P xn }^ =1 
and Q = {Q xn }%Li, where P xn and Q xn represent the 
product distributions of P and Q on X n . Then 



limlimsup-P4(P n ||Q„) = D(P\\Q). 
Proof: We will first prove 

limlimsup-P^(P n ||Q„) < J(P;Q) 

e— >0 n — ^oo ?l 



(10) 



Consider any A > I(P; Q). Let us define the following 
set 

1 ,„ ^n(x) 



< A 



(11) 



(12) 



Let <p n : X n -> [0, 1], n G N, such that 

'P n (x) ifxeA(A), 
otherwise. 

From (O it easily follows that 

lim Pi'M„(A)} = 1. (13) 

n— »oc 

Thus from our construction of <j) n , (TT2l) . it follows that 

lim V 0„(x) = lim Pr{A,(A)} = 1. (14) 

n— loo — ' n— >oo 
x£X™ 

Using dT2b and ( [Pfl ) observe that for n large enough 

lim lim sup — D e (P n \\Q n ) < lim sup - log max ^" ^\ 

E-KI n ->oo n n^oo U xGyt„(A) Q n (X) 

a 

< A, 

where a follows from ( fTTT i and ( fT2l . 
We now prove the other side, i.e., 

lim limsup-r4(P n ||Q„) > J(P;Q). 

Consider any 7 < J(P; Q). For every n£ff, let us define 
the following the set 



(15) 



From (0 it follows that there exists 77 G (0, 1], such that 
lim sup Pr^ (7)} =V- (16) 

n— >-oo 

Since Pr{A„(7)} + Pr{.A£(7)} = 1, for every n G N, 
we have 

liminf Pr{^ (7)} = 1 — 77. (17) 

n— >oo 

For every e £ (0, 77), let us define a sequence of positive 
functions {(pn}^^ sucn mat f° r ever y n G N 

</>„ : A" 1 -> [0,1], ^.(x) < P„(x),Vx G A" 1 

and ^ n (x) > 1 -e. (18) 

We now claim that for large enough n, Supp(</> n ) n 
•^1(7) 7^ 0- To prove this claim, suppose that Supp(0„)n 
-4™ (7) — 4>- This would further imply that 

liminf 0„(x) < liminf Pr{.4£ (7)} 

n— >oo — J n— too 



xEX r 



= 1- 7] 
b 

<l-e, 



(19) 



where a follows from (fTTJ and b follows because e < r\. 
Notice that ( fT9b contradicts (IT8T >. 
Thus for n large enough, 

l-e< J2 <M X ) + J2 

<Pr{^( 7 )}+ ]T 0„(x). 

xe^„( 7 ) 

By rearranging the terms in the above equation we get 

l- £ -Pr{^( 7 )}< 0«(x). (20) 

xe^„(7) 

Taking lim sup on both sides of d20b . we have 
limsup 0n(x) > 1 — e — liminf Pr{./1^(7)} 

> J7 - e. (21) 

dJTJ follows from ( fTTI i. Now notice the following set of 
inequalities for large enough n 

i> E p «w 

xG^l„(7) 
> 2 2 " 7 Q™( X ) 

xe^„( 7 ) 

| 2 („ Xx ^ log -«) ^ ^ i(x) (22) 

xG^l„(7) 

where a follows from (TT3T >; b follows from the fact that 
for every x S .A™ (7), 

0n(x) <^n(x) ^ 0n(x) 

< max - ; , < max 



<2„(x) xe^"( 7 ) Qn(x) xe^~ Q„(x) 

By taking log on both sides of (1221 and rearranging the 
terms we get 

1 , 0n(x) 1 J, I \ 

max - log > 7 + - log > (pnW- 

^ ' x6^„( 7 ) 

Taking limsup on both sides of the above equation we 
have 

lim sup max — log < ^r—, \ > 7 + lim sup — log <£ 

xe^„( 7 ) 

> 7 (23) 

where ( 1231 follows from (12TV Notice that ( |23l is true for 
every 0„ satisfying ( TT~8b - Thus 



IV. Source coding with coded state side 

INFORMATION AVAILABLE AT THE DECODER 

Let (X n , Y n ) ~ Px«y», with range x ^)", where 

(x n ,Y n ) := [(jf 1 ,y 1 ) > (jf 2 ,y 2 ) J ...,(A Tl ,y n )]. 

The n-shot source coding with coded side information 
available at the decoder is formulated as follows. We first 
define two sets of integers 



Atf J ={!>••• ,2*S—Cx-) }> 
7W£ 2) ={!,... ,^-( r )} 



(25) 
(26) 



called the codes. Choose arbitrary mappings : ^ n — > 
M ( n ] (encoderl) and : y n -)• 7Wi 2) (encoder2). We 
call 



^_c„c(^) log|M 



(2) 



n n 
the coding rates of the encoder 1 and encoder 2, respec- 
tively. The decoder d n : Mn^ x — > X n receives 
two outputs e^^x) and e£\y) from the two encoders 
and tries to reconstruct the original source output x. Thus 
the probability of error for this task is defined as 

P e " := Pr{X n ^ X n }, 

where X n = d n (e { n\x n ), e { n\Y n )). Note here that the 



encoders e« and e„ do not cooperate with each other. 
We call the triplet (en\en ,d n ) of two encoders and 



one decoder with the two codes in (125t and d26t and the 
error probability e the (n, 2^-e M ( Jf ") ) 2^-^ Y "\ e) n- 
shot code. 

In this coding system we wish to minimize the two 



coding rates and 



such that the 



probability of error is less than e. 

Definition 5: (One-shot e achievable rate pair) Let 
(X,Y) ~ Pxy, with range X x y. A one-shot rate 
pair (i?i,i?2) is called £ achievable if and only if there 
exists a (1, 2 ld - m ^ x \ 2 la --^^ Y \ e) one-shot code such 
Wthat Pr{X ^X}<e, £|_ cnc (X) < R 1 and t%_ eTLC {Y) < 
R 2 - 

Definition 6: (Asymptotically achievable rate pair) A 
rate pair (R%, R2) is asymptotically achievable if and only 
if there exists (n, 2 £ d-c„c(^") j 2 ld ~^ Yn \ e) code such 
that Yv{X n ± X n } < e, 



limsup -I£,(P„ I \Q n ) > 7. 



(24) 



Since (l24l is true for every e £ (0, 77), the result will hold 
true for e \ 0. 

( ITDl easily follows from the law of large numbers and 
©. This completes the proof. ■ 



and 



lim lim sup ■ 

€ ^0 7j — vnn 



lim lim sup ■ 

^0 n — inn 



<Ri 



Theorem 1: Let (X, Y) ~ Pxy, with range X x y. 
For the error e G (0, 1). The following one-shot rate 
region for source coding of X with a helper observing 
Y is achievable 

^d-c„c(^) > H^(X\U) - log(e - ei), 

+ log[- ln(ei - en - 2e 1 2 1 )] 

for some conditional pmf Pmy> where e\ < e and en is 
such that 

en + 2e u < £l and D e £ (Puy\\Pu x Py) > 0. (27) 

Proof: The techniques used in the proof here are 
motivated from IT3l Lemma 4.3]. Fix a conditional 
probability mass function Pjj\y an d let Pu(u) = 
^yey Py(y)Pu\Y{u\y). Choose en such that the con- 
ditions in < f27b are satisfied. Notice that such a choice 
of en always exists because D^ 1 (Puy\\Pjj x Py) is 
a decreasing function of E\\. Let Q G B £ll (Pjjx) and 
4> G B eil {P UY ) be such that 

m ix (X\U) = logmax|Supp(Q(X|£7 = u))\ (28) 




log- 



(u,y):P UY (u,y)>Q Pu(u)P Y (y)' 



ifPy(2/)>0 J 

otherwise. 



(29) 



(30) 



Notice that the triplet (X, Y, U) satisfy the following 

X^Y^U. (31) 

For more details on ( |3TT > see iTPJl equation 4.4]. For every 
(u, y) GU x y, let p be a mapping such that 

9(u,y) ■= ^ p x\Y(x\y)I(x,u), (32) 

where I(.t,w) for every (x,u) G ^ x W is defined as 
follows 



u) 



1 if (x,it) ^ Supp(Q), 
otherwise. 



(33) 



Define the following set 

T:={(u,y)eUxy:g(u,y)<ei}. (34) 

Random code generation: Randomly and indepen- 
dently assign an index i € [1 : 2 e<1 -<= n ^ x ' > ] to every 
realization x G X. The realizations with the same index 



i form a bin B(i). Randomly and independently generate 
2 £ d-c„c(X) realizations u(k), k G [1 : each 
according to Pjj- 

Encoding: If the encoder 1 observes a realization x G 
#(£), then the encoder 1 transmits i. For every realization 
y G y the encoder 2 finds an index k such that (u(k),y) G 
J". For the case when there are more than one such index, 
it sends the smallest one among them. If there is none, it 
then sends k = 1. 

Decoding: The receiver finds the unique x' G B(i) 
such that (x',u(k)) G Supp(Q). 

Probability of error analysis: Let Mi and M 2 be 
the chosen indices for encoding X and Y. The error in 
the above mentioned encoding decoding strategy occurs 
if and only if one or more of the following error events 
occur 



}■ 



E 1 = {(U(m 2 ),Y) £ F, Vm 2 G [l : 2*S-~ .00 
£ 2 = {(X,(7(Af 2 ))^Supp(Q)}, 

£ 3 = {3a;' G B(mi) : (a:', tf(M 2 )) G Supp(Q), a/ ^ X} . 

For more details on error events see lfP7] Lemma 4]. The 
probability of error is upper bounded as follows 

Pr{£} < Pr{£i} + Pr{^ n E 2 } + Pr{E 3 \X G B(l)}. 

(35) 

We now calculate Pt{E\} as follows 



Pr{Ei} 

= J2Py(y)P^\(u(m 2 ),y) i -F,Vm 2 G 



yey 



1 : 2'3- 



= 00 



= ^Py(y) 1- X! P ^ M ) 
yey \ u:(u,;/)e^ 



xiV) 



< 1 - 51 <K U > !/) + e 
<eii+Pr{(C/,y) ^ J 7 } 



ii:(ii,y)€J r 



-2'd- 



c (i') 2 -Djll(J> U y||J> [ ,XP y ) 



where a follows because U(l), . . . , f7(2 d-«"»00) are 
independent and subject to identical distribution Pjj; b 
follows from Q, d29l and (f30b ; c follows from the 
inequalities (1 - x) y < e~ xy (0 < x < l,y > 0) and 
e -*v < l - y + a; (a; > 0, < ?/ < 1) and ([30]>; d is true 



because of the following arguments 

i-£n< Y <t>(u,y) 

(u,y)£Uxy 

= Y ( f ) ( u >y)+ Y <Pi u ^v) 

(«,j/)ej rc («,y)eJ r 
<Pr{7 c } + Y, ^ U 'V) 

<Pr{(C/,y) i ^ (36) 

where a and b both follow from the fact that <p(u, y) G 
B £ii (P(7y)- By rearranging the terms in (l36l we get 

1- £ 0(u,y) <en+PT{(U,Y) i T}. (37) 
(«,3/)e^ 

We now calculate Pr{(U,Y) £ J 7 } as follows 

Pr{(C/, Y) J"} = Pr{. 9 (t/,r) > 4} 

<e^Euy(ff(^y)) 

<£n Y p UY{u,y)g(u,y) 
(u,y)euxy 

= £ ii Y p vv{u,y) 
(u,y)euxy 

Y Px\Y{x\y)I{x,u) 

x£X 
-i v— v 

= £n 2 2^ PxuY(x,u,y) 

(x,u,y)eXxUxy 
(«,x)^Supp(Q) 

= £n 5 51 Pc/x(w,a;) 

(u,x)£Supp(Q) 



— fc ll> 



(38) 



where a follows from Markov's inequality; b follows from 
(l32l and c follows because of the following arguments 

1 - en < 2^ Q(u,x) 
(u,x)euxx 

< Y p ux(u,x), (39) 
(«,x)eSupp(Q) 



where a follows from (1281 1 and b follows because 
Q(u,x) < Pux{u,x), for every (u, x) G U x X. By 
rearranging the terms in ((39} we get 

1 - Y Pux{u,x) < En. 
(«,x)eSupp(Q) 



The second term in (1351 1 is calculated as follows 

Pr{Ef n E 2 } 

= Y Pxur(x,u,y) 

(x,u,y)eXxyxU 
(u,2/)e.F,(u,x)£Supp(Q) 

= Y p UY(u,y) Y p x\ Y {x\y) 

(«,2/)e.F x:(x,u)£Supp(Q) 



(40) 



where the last inequality follows from (l34l i. From (l37t . 
( 1381 ) and (gOjl it follows that 

Pr{Bi} + Pr{^n^j} 



< en + 2e 1 2 1 + e" 



Let 



£i > £11 + ^£fi + e . (41) 

It now easily follows that 

^_ enc (F)>log[-ln(£ 1 -£ 1 i-24 1 )] 
+ D^(P uy \\PuxPy). 



Finally, the third term in (135) is calculated as follows 

Pr{£ 3 } = Pr{£ 3 |6(l)} 

= Y Pr{(X,tO = MI*eS(i)} 



Pr ^ x : x G 6(l)and Q(V, it) > 
la G B(1),(X,U) = (x,u) 



< Y p xu(x,u) Y p ^'eB(l)} 

(x.u)eXxU x'jtx 

Q(x',u)>0 

<2- £ S-c„c(^) P xu (x,u)max V 1 

new ^— ' 

(l.ujEA'xW x:Q(x,m)>0 

V Pxc/(a:,u)max|Supp(Q(X|?7 = ti))| 

(42) 

= max |Supp(Qpf |[/ = it))|, 

new 

where a follows because Q(X = x\U = u) := p^'^j 1 
and Q(X = x\U = u) := if P v {u) = 0. Thus from 
(HTt and (|42j it follows that 

Pr{£} < £j +2-^-c»c(^) max |Supp(Q(X|C/ = u))|. 



Let 

ei + 2~^-«.cW max |Supp(Q(X|{7 = u))| < e. 
It now easily follows that 

i^_ cnc (X) > H^(X\U) -log(s - £l ). 

This completes the proof. ■ 
The asymptotic optimality of the rate region obtained 
in Theorem [T] is an immediate consequence of Definition 
[6] Lemma Q] and [3] 

V. Conclusion and Acknowledgements 

We proved that smooth max divergence and smooth 
max conditional Renyi entropy can be used to obtain one- 
shot achievable rate region for source coding when coded 
side information is available at the decoder. Furthermore, 
we showed that asymptotically this region coincides with 
the rate region as derived by Wyner in ifPJI . 

The author gratefully acknowledges the helpful discus- 
sions with Mohit Garg and Sarat Moka. 
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